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Quantitative computational models play an increasingly important role in modern biology. Such 
models typically involve many free parameters, and assigning their values is often a substantial 
obstacle to model development. Directly measuring in vivo biochemical parameters is difficult, and 
collectively fitting them to other experimental data often yields large parameter uncertainties. Nev- 
ertheless, in earlier work we showed in a growth-factor-signaling model that collective fitting could 
yield well-constrained predictions, even when it left individual parameters very poorly constrained. 
We also showed that the model had a 'sloppy' spectrum of parameter sensitivities, with eigenvalues 
roughly evenly distributed over many decades. Here we use a collection of models from the literature 
to test whether such sloppy spectra are common in systems biology. Strikingly, we find that every 
model we examine has a sloppy spectrum of sensitivities. We also test several consequences of this 
sloppiness for building predictive models. In particular, sloppiness suggests that collective fits to 
even large amounts of ideal time-series data will often leave many parameters poorly constrained. 
Tests over our model collection are consistent with this suggestion. This difficulty with collective fits 
may seem to argue for direct parameter measurements, but sloppiness also implies that such mea- 
surements must be formidably precise and complete to usefully constrain many model predictions. 
We confirm this implication in our growth-factor-signaling model. Our results suggest that sloppy 
sensitivity spectra are universal in systems biology models. The prevalence of sloppiness highlights 
the power of collective fits and suggests that modelers should focus on predictions rather than on 
parameters. 



Non-technical Summary: Dynamic systems biol- 
ogy models typically involve many kinetic parameters, 
the quantitative determination of which has been a se- 
rious obstacle to using these models. Previous work 
showed for a particular model that useful predictions 
could be extracted from a fit long before the experimen- 
tal data constrained the parameters, even to within or- 
ders of magnitude. This was attributed to a 'sloppy' 
pattern in the model's parameter sensitivities; the sensi- 
tivity eigenvalues were roughly evenly spaced over many 
decades. Consequently, the model behavior depended ef- 
fectively on only a few 'stiff' parameter combinations. 
Here we study the converse problem, showing that direct 
parameter measurements are very inefficient at constrain- 
ing the model's behavior. To yield effective predictions 
such measurements must be very precise and complete; 
even a single imprecise parameter often destroys predic- 
tivity. We also show here that the characteristic sloppy 
eigenvalue pattern is reproduced in sixteen other diverse 
models from the systems biology literature. The appar- 
ent universality of sloppiness suggests that predictions 
from most models will be very fragile to single uncer- 
tain parameters and that collective parameters fits can 
often yield tight predictions with loose parameters. To- 
gether these results argue that focusing on parameter val- 
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ues may be a very inefficient route to useful models. 



Dynamic computational models are powerful tools for 
developing and testing hypotheses about complex biolog- 
ical systems [1 , 2,3. ft has even been suggested that such 
models will soon replace databases as the primary means 
for exchanging biological knowledge [3]. A major chal- 
lenge with such models, however, is that they often pos- 
sess tens or even hundreds of free parameters whose val- 
ues can significantly affect model behavior 0[6]. While 
high-throughput methods for discovering interactions are 
wcll-dcvclopcd [7 , high-throughput methods for measur- 
ing biochemical parameters remain limited [8]. Further- 
more, using values measured in vitro in an in vivo ap- 
plication may introduce substantial inaccuracies [51 HO] ■ 
On the other hand, collectively fitting parameters [TT1IT2"] 
by optimizing the agreement between the model and 
available data often yields large parameter uncertain- 
ties [131 HU [T5]. In approaches typically more focused 
on steady-state distributions of fluxes in metabolic net- 
works, metabolic control analysis has been used to quan- 
tify the sensitivity of model behavior with respect to pa- 
rameter variation |16j . and flux-balance analysis and re- 
lated techniques have probed the robustness of metabolic 
networks [T7l [18]. 

One way to cope with the dearth of reliable parame- 
ter values is to focus on predictions that are manifestly 
parameter- independent [19j . but these are mostly quali- 
tative. An alternative is not to forsake quantitative pre- 
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dictions, but to accompany them with well-founded un- 
certainty estimates based on an ensemble of parameter 
sets statistically drawn from all sets consistent with the 
available data [5U]. (Uncertainties in the model structure 
itself may be important in some cases. Here we focus on 
parameter uncertainties, as they are often important on 
their own.) 

Brown et al. took this approach in developing a 
computational model of the well-studied growth-factor- 
signaling network in PC12 cells [21 . They collectively 
fit their model's 48 parameters to 68 data points from 
14 cell-biology experiments (mostly Western blots). Af- 
ter the fit, all 48 parameters had large uncertainties; their 
95% confidence intervals each spanned more than a factor 
of 50. Surprisingly, while fitting this modest amount of 
data did not tightly constrain any single parameter value, 
it did enable usefully tight quantitative predictions of be- 
havior under interventions, some of which were verified 
experimentally. 

In calculating their uncertainties, Brown et al. found 
that the quantitative behavior of their model was much 
more sensitive to changes in certain combinations of pa- 
rameters than others. Moreover, the sensitivity eigen- 
values were approximately equally spaced in their loga- 
rithm, a pattern deemed 'sloppy'. Such sloppy sensitivi- 
ties were subsequently seen in other multi-parameter fit- 
ting problems, from interatomic potentials |22j to sums 
of exponentials [25] . The fact that sloppiness arises in 
such disparate contexts suggests that it may be a univer- 
sal property of nonlinear multi-parameter models. (Here 
the term 'universal' has a technical meaning from statis- 
tical physics, denoting a shared common property with a 
deep underlying cause; see [23J . Universality in this sense 
does not imply that all models must necessarily share the 
property.) 

In this work, we begin by empirically testing seven- 
teen systems biology models from the literature, examin- 
ing the sensitivity of their behavior to parameter changes. 
Strikingly, we find that Brown et aVs model is not unique 
in its sloppiness; every model we examine exhibits a 
sloppy parameter sensitivity spectrum. (Thus, in the 
models we've examined sloppiness is also universal in 
the common English sense of ubiquity.) We then study 
the implications of sloppiness for constraining parame- 
ters and predictions. We argue that obtaining precise 
parameter values from collective fits will remain difficult 
even with extensive time-series data, because the behav- 
ior of a sloppy model is very insensitive to many param- 
eter combinations. We also argue that, to usefully con- 
strain model predictions, direct parameter measurements 
must be both very precise and complete, because sloppy 
models are also conversely very sensitive to some param- 
eter combinations. Tests over our collection of models 
support the first prediction, and detailed analysis of the 
model of Brown et al. supports the second contention. 

Sloppiness, while not unique to biology, is particu- 
larly relevant to biology, because the collective behav- 
ior of most biological systems is much easier to mea- 



sure in vivo than the values of individual parameters. 
Much work has focused on optimizing experimental de- 
sign to best constrain model parameters with collective 
fits [211I2I3I2S]. We argue against this focus on parameter 
values, particularly when our understanding of a system 
is tentative and incomplete. Concrete predictions can be 
extracted from models long before their parameters are 
even roughly known |21j . and when a system is not al- 
ready well-understood, it can be more profitable to design 
experiments to directly improve predictions of interesting 
system behavior |27j rather than to improve estimates of 
parameters. 



I. RESULTS 

A. Systems Biology Models have Sloppy 
Sensitivity Spectra 

Our collection of 17 systems biology models [2] [21] [25] 
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was drawn primarily from the BioModels database [4"2"] . 
an online repository of models encoded in the Systems 
Biology Markup Language (SBML) [43 . The collected 
models encompass a diverse range of biological systems, 
including circadian rhythm, metabolism, and signaling. 
All the models arc formulated as systems of ordinary dif- 
ferential equations, and they range from having about 
ten to more than two hundred parameters. In most cases, 
these parameters were not systematically fit or measured 
in the original publication. 

We quantified the change in model behavior as parame- 
ters 9 varied from their published values 9* by the average 
squared change in molecular species time courses: 

2 (ff] = i v 1 f T ' [ y a , c (0,f)-y a , c (flV) 1 2 d , 
x 1 1 2N C N S ^T C J [ a s \ ' 

a kind of continuous least-squares fit of parameters 9 
to 'data' simulated from published parameters 9*. Here 
Us,c(9, t) is the time course of molecular species s given 
parameters 9 in condition c, and T c is the 'measurement' 
time for that condition. We took the species normal- 
ization o~ s to be equal to the maximum value of species 
s across the conditions considered; other consistent nor- 
malizations yield the same qualitative conclusions. 

For each model, the sum in Equation [T] runs over all 
molecular species in the model and (except where infca- 
sible) over all experimental conditions considered in the 
corresponding paper — an attempt to neutrally measure 
system behavior under conditions deemed significant by 
the original authors. (The total number of conditions and 
species are denoted by N c and N s , respectively.) SBML 
files and SloppyCell [H] scripts for all models and condi- 
tions are available in Dataset SI. 

To analyze each model's sensitivity to parameter vari- 
ation, we considered the Hessian matrix corresponding 
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FIG. 1: Parameter sensitivity spectra 
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Subfigure A illustrates the quantities we calculate from H x , while subfigures B and C show that all the models we examined 
have sloppy sensitivity spectra. 

2 o 

A: Analyzing H x corresponds to approximating the surfaces of constant model behavior change (constant x ) as ellipsoids. 
The width of each principal axis is proportional to one over the square root of the corresponding eigenvalue. The inner ellipsoid's 
projection onto and intersection with the log 9i axis are denoted Pi and I\ , respectively. 
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B: Plotted are the eigenvalue spectra of H x for our collection of systems biology models. The many decades generally spanned 
indicate the ellipses have very large aspect ratio. (The spectra have each been normalized by their largest eigenvalue. Not all 
values are visible for all models.) 

C: Plotted is the spectrum of I/P for each parameter in each model in our collection. Generally very few parameters have 
I/P w 1, suggesting that the ellipses are skewed from the bare parameter axes. (Not all values are visible for all models.) 
The models are ordered by increasing number of free parameters and are: (a) eukaryotic cell cycle [28], (b) Xenopus egg cell 
cycle [29] . (c) eukaryotic mitosis [3D], (d) generic circadian rhythm |31| . (e) nicotinic acetylcholine intra-receptor dynamics [32] . 
(f) generic kinase cascade [33], (g) Xenopus Wnt signaling [34], (h) Drosophila circadian rhythm [35], (i) rat growth-factor 
signaling [21] . (j) Drosophila segment polarity [36] . (k) Drosophila circadian rhythm [37], (1) Arabidopsis circadian rhythm [2], 
(m) in silico regulatory network [25], (n) human purine metabolism [38], (o) E. coli carbon metabolism [39| . (p) budding yeast 
cell cycle [40] . (q) rat growth-factor signaling [41] . 
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We took derivatives with respect to log 9 to consider rel- 
ative changes in parameter values, because biochemical 
parameters can have different units and widely varying 
scales. Analyzing H x corresponds to approximating the 
surfaces of constant model behavior deviation (as quan- 
tified by x 2 ) to be iV p -dimensional ellipsoids, where N p 
is the number of parameters in the model. Figure [l]A_ 



schematically illustrates these ellipsoids and some of their 
characteristics. (Details of calculating H x and related 
quantities are found in Methods. Dataset SI includes 
H x for each model.) 

The principal axes of the ellipsoids are the eigenvectors 
of H x , and the width of the ellipsoids along each princi- 
pal axis is proportional to one over the square root of the 
corresponding eigenvalue. The narrowest axes are called 
'stiff', and the broadest axes 'sloppy' [20]. The eigen- 
value spectra for the models in our collection are shown 
in Figure |Tj3 (each normalized by its largest eigenvalue) . 
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In every case, the eigenvalues span many decades. All 
but one span more than 10 6 , indicating that the slop- 
piest axes of the ellipsoids illustrated in Figure [T}\ are 
generally more than one thousand times as long as the 
stiffcst axes. In each spectrum the eigenvalues are also 
approximately evenly spaced in their logarithm; there is 
no well-defined cutoff between 'important' and 'unimpor- 
tant' parameter combinations. 

The Hessian matrix is a local quadratic approxima- 
tion to the generally nonlinear \ 2 function. Principal 
component analysis of extensive Monte Carlo runs in the 
Brown et al. model, however, indicates that the sloppi- 

2 , ~ 

ness revealed by H x is indicative of full nonlinear \ 
function [20] . 

Along with their relative widths, the degree to which 
the principal axes of the ellipsoids are aligned to the bare 
parameter axes is also important. We estimated this by 
comparing the ellipsoids' intersections Ii with and pro- 
jections Pi onto each bare parameter axis i. If Ii/ Pi = 1 
then one of the principal axes of the ellipsoids lies along 
bare parameter direction i. Figurejlp plots the I/P spec- 
trum for each model. In general, very few axes have 
I/P « 1; the ellipses are skewed from single parameter 
directions. 

Naively, one might expect the stiff eigenvectors to em- 
body the most important parameters and the sloppy di- 
rections to embody parameter correlations that might 
suggest removable degrees of freedom, simplifying the 
model. Empirically, we have found that the eigenvectors 
often tend to involve significant components of many dif- 
ferent parameters; plots of the five stiff est eigenvectors 
for each model are in Supporting Text SI. This is un- 
derstandable theoretically; the nearly-degenerate sloppy 
eigenvectors should mix, and the stiff eigenvectors can in- 
clude arbitrary admixtures of unimportant directions to 
a given important parameter combination. (Indeed, in 
analogous random-matrix theories the eigenvectors are 
known to be uncorrelated random vectors |45j.) While 
the relatively random eigenvectors studied here may not 
be useful in guiding model reduction, more direct explo- 
rations of parameter correlations have yielded interesting 
correlated parameter clusters [4"rJ] , 

These characteristic parameter sensitivities that evenly 
span many decades and are skewed from bare parameter 
axes define a 'sloppy' model [2D]. Figures[Tj3 and[lp show 
that every model we have examined has a sloppy sensi- 
tivity spectrum. Next we discuss some broad questions 
about the relation between model predictions, collective 
fits, and parameter measurements and see how the sloppy 
properties of these models may suggest answers. 



B. Consequences of Sloppiness 

The difficulty of extracting precise parameter values 
from collective fits in systems biology modeling is well- 
known |26j . Sloppiness offers an explanation for this and 
predicts that it will be true even for fitting to complete 




FIG. 2: Sloppiness and uncertainties 

As in Figure [IJ\, the contours represent surfaces of constant 
model behavior deviation. The clouds of points represent pa- 
rameter set ensembles. 

A: Collective fitting of model parameters naturally constrains 
the parameter set ensemble along stiff directions and allows 
it to expand along sloppy directions. The resulting ensemble 
may be very large, yet encompass little variation in model 
behavior, yielding small prediction uncertainties despite large 
parameter uncertainties. (Si denotes the 95% confidence for 
the value of 9\ .) 

B: If all parameters are directly measured to the same preci- 
sion, the parameter set ensemble is spherical. The measure- 
ment precision required for well-constrained predictions is set 
by the stiffest direction. 

C: If one parameter (here 62) is known less precisely than the 
rest, the cloud is ellipsoidal. If not aligned with a sloppy di- 
rection, the cloud will admit many model behaviors and yield 
large prediction uncertainties. (Note that the aspect ratio of 
the real contours can be greater than 1000.) 



data that the model can fit perfectly. In a collective fit, 
the parameter set ensemble samples from all sets of pa- 
rameters for which the model behavior is consistent with 
the data. Because sloppy models are very insensitive to 
parameter combinations that lie along sloppy directions, 
the parameter set ensemble can extend very far in those 
directions, as illustrated schematically in Figure [2jV. As 
a result, individual parameters can be very poorly deter- 
mined (e.g., confidence interval indicated by St in Fig- 
ure [2]A_) . Below we discuss a test of this prediction over 
all the models in our collection. 

Unless one has direct interest in the kinetic constants 
for the underlying reactions, uncertainties in model pre- 
dictions are generally more important than uncertainties 
in model parameters. The parameter set ensemble illus- 
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trated in Figure [2]A_ yields large uncertainties on indi- 
vidual parameters, but can yield small uncertainties on 
predictions. While the fitting process allows the ensemble 
to expand along sloppy directions, the fit naturally con- 
strains the ensemble along stiff directions, so that model 
behavior varies little within the ensemble, and predic- 
tions can be consequently tight. 

Direct parameter measurements, on the other hand, 
will have uncertainties that are uncorrelated with the 
model's underlying stiff and sloppy directions. For exam- 
ple, if all parameter measurements are of the same pre- 
cision, the parameter set ensemble is spherical, as illus- 
trated in Figure |2}3. For tight predictions, this ensemble 
must not cross many contours, so the required precision 
is set by the stiffest direction of the model. Consequently, 
high precision parameter measurements are required to 
yield tight predictions. Moreover, these measurements 
must be complete. If one parameter is known less pre- 
cisely, the parameter set ensemble expands along that 
parameter axis, as illustrated in Figure[2p. If that axis is 
not aligned with a sloppy direction, model behavior will 
vary dramatically across the parameter set ensemble and 
predictions will have correspondingly large uncertainties. 
Below we discuss tests of both these notions, exploring 
the effects of direct parameter measurement uncertainty 
on predictions of a particular model. 



1. Parameter Values from Collective Fits 

Does the sloppiness of these models really prevent one 
from extracting parameters from collective fits? Here we 
discuss a test of this prediction using an idealized fitting 
procedure. 

Our x 2 measure of model behavior change (Equation[l]) 
corresponds to the cost function for fitting model pa- 
rameters to continuous time-series data that the model 

2 

fits perfectly at parameters 0* ; H x is the corresponding 
Fisher information matrix (Equation [2| . We used this 
idealized situation to test the prediction that collective 
fits will often poorly constrain individual parameters in 
our collection of sloppy models. 

We defined the relative 95% confidence interval size 
as the ratio between parameter i at the upper and lower 
extremes of the interval, minus one. (A parameter value 
constrained after the fit to lie between 10 and 1000 would 
have £ ~ 100, while one constrained between 1.0 and 1.5 
would have £ = 0.5.) We assumed 100 times as many 
data points (each with 10% uncertainty) as the number 
of parameters in each model. Figure [3] shows histograms 
of the quadratic approximation to £ for each parame- 
ter in each model after fitting such data. (See Meth- 
ods.) For most of the models, the figure indicates that 
such fitting leaves many parameters with greater than 
100% uncertainty (S > 1). Indeed, even fitting this large 
amount of ideal data can leave many parameter values 
very poorly determined, as expected from the sloppiness 
of these models and our discussion of Figure [2]A 




Relative confidence 
interval size 



FIG. 3: Fitting parameters to idealized data 
Shown are histograms of the relative confidence interval size 
E for each parameter in each model of our collection, after 
fitting 100 times as many time-series data points (each with 
10% uncertainty) as parameters. In most cases a large number 
of parameters are left with greater than 100% uncertainty. (A 
parameter constrained with 95% probability to lie between 1 
and 100 would have E w 100.) 
Labels are as in Figure [l] 



The fact that nonlinear multiparameter models often 
allow a wide range of correlated parameters to fit data 
has long been appreciated. As one example, a 1987 paper 
by Brodersen et al. on ligand binding to hemoglobin and 
albumin empirically found many sets of parameters that 
acceptably fit experimental data, with individual param- 
eter values spanning huge ranges |13j . Our sloppy model 
perspective ( [2Ql EH |23] , Figure 1) shows that there is a 
deep underlying universal pattern in such least-squares 
fitting. Indeed, an analysis of the acceptable binding pa- 
rameter sets from the 1987 study shows the same charac- 
teristic sloppy eigenvalue spectrum we observed in Fig- 
ure \lp (Supporting Text S5). 



2. Predictions from Direct Parameter Measurements 

Figures[2)3 and[2p suggests that direct parameter mea- 
surements must be both precise and complete to usefully 
constrain predictions in sloppy systems. Here we discuss 
a test of this notion in a specific model. 

We worked with the 48-parameter growth-factor- 
signaling model of Brown et al, shown schematically in 
Figure [2T]. The parameters in this model were origi- 
nally collectively fit to 14 time-scries cell-biology experi- 
ments. We focused on this model because it is instructive 
to compare our results concerning direct parameter mea- 
surements with prior results from collective fitting. For 
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FIG. 4: Parameter and prediction uncertainties 
A: Our example prediction is for ERK activity upon EGF 
stimulation given PI3K inhibition in this 48-parameter model 
of growth-factor-signaling in PC12 cells |21| . 
B: Shaded regions are 95% confidence intervals calculated via 
exhaustive Monte Carlo for our example prediction given var- 
ious scenarios for constraining parameter values. 
C: Plotted is the relative size £ of the 95% confidence interval 
for each parameter. 

The scenarios represented are: (red, squares) all model pa- 
rameters individually measured to high precision, (blue, trian- 
gles) all parameters precisely measured, except one estimated 
to low precision, (yellow, circles) all parameters collectively fit 
to 14 real cell-biology experiments. Precisely measured indi- 
vidual parameter values enable a tight prediction (B: middle 
red band), but even one poorly known parameter can destroy 
predictive power (B: wide blue band). In contrast, the collec- 
tive fit yields a tight prediction (B: tightest yellow band) but 
only very loose parameter constraints (C: circles). The large 
parameter uncertainties from the collective fit (C: circles) cal- 
culated here by Monte Carlo are qualitatively similar to those 
seen in the linearized fit to idealized data (Figure [3] model 
(i)). (For clarity, the dashed red lines trace the boundary of 
the red confidence interval.) 



our analysis, we assumed that hypothetical direct param- 
eter measurements would be centered about the original 
best-fit values. 

One important test of the model was a prediction of 
the time-course of ERK activity upon EGF stimulation, 
given inhibition of the PI3K branch of the pathway. The 
yellow shaded region in Figure [4j3 shows the uncertainty 
bound on this prediction from the original collective fit, 
calculated by exhaustive Monte Carlo [2TJ. The tight- 
ness of this prediction is remarkable considering the huge 
uncertainties the collective fit left in the individual pa- 
rameter values (yellow circles in Figure |4p). Not a single 
parameter was constrained to better than a factor of 50. 

How precise would direct parameter measurements 
have to be to yield as tight a prediction as the collec- 
tive fit? For this prediction, the PI3K branch (inhibited) 
and C3G branch (NGF-dependent) of the pathway are ir- 
relevant in the model; the remaining reactions involve 24 
parameters. To achieve the red prediction in Figure [4}3, 
all 24 involved parameters must be measured to within 
a factor of plus or minus 25% (Figure |4p, red squares). 
With current techniques, measuring even a single in vivo 
biochemical parameter to such precision would be a chal- 
lenging experiment. Such high precision is required be- 
cause, as illustrated in Figure[2}3, the measurements need 
to constrain the stiffest combination of model parame- 
ters. 

What if a single parameter is left unmeasured? For 
example, consider high precision measurements of 23 of 
the 24 involved parameters, all but the rate constant for 
the activation of Mek by Raf 1 . For this unmeasured pa- 
rameter, we assumed that an informed estimate could 
bound it at 95% confidence to within a total range of 
1000 {e.g., between Is" 1 and 1000s" 1 ). The resulting 
prediction (blue in Figure |4|3) has very large uncertainty 
and would likely be useless. Note that these hypothet- 
ical measurements constrain every individual parameter 
value more tightly than the original collective fit (blue 
triangles versus yellow circles in Figure |4p) , yet the pre- 
diction is much less well-constrained. Neither this pa- 
rameter nor this prediction is unique. Uncertainty for 
this prediction is large if any one of about 18 of the 24 in- 
volved parameters is unmeasured (Supporting Text S2). 
Furthermore, other possible predictions in this model are 
similarly fragile to single unmeasured parameters (Sup- 
porting Text S3). 

To usefully constrain Brown et al.'s model, direct pa- 
rameter measurements would need to be both precise 
and complete. By contrast, collective parameter fitting 
yielded tight predictions with only a modest number 
of experiments. These results are expected given the 
model's sloppiness. 



II. DISCUSSION 

By examining seventeen models from the systems biol- 
ogy literature [2JJ [25J HH HO 133 ED 133 131 131 [S3 H3 
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[571 (Ml GSO HOI SI], we showed that their parameter sensi- 
tivities all share striking common features deemed 'slop- 
piness'; the sensitivity eigenvalues span many decades 
roughly evenly (Figure [Tj3), and tend not to be aligned 
with single parameters (Figure [lp). We argued that 
sloppy parameter sensitivities help explain the difficulty 
of extracting precise parameter estimates from collective 
fits, even from comprehensive data. Additionally, we 
argued that direct parameter measurements should be 
inefficient at constraining predictions from sloppy mod- 
els. We then showed that collective parameter fits to 
complete time-series data do indeed yield large parame- 
ter uncertainties in our model collection (Figure [3| . Fi- 
nally, we confirmed for the 48-parameter signaling model 
of Brown et al. |21j that direct parameter measurements 
must be formidably precise and complete to usefully con- 
strain model predictions (Figure [4]). 

What causes sloppiness? (1) Fundamentally, slop- 
piness involves an extraordinarily singular coordinate 
transformation in parameter space between the bare pa- 
rameters natural in biology {e.g., binding affinities and 
rate constants) and the cigcnparamctcrs controlling sys- 
tem behavior, as discussed in |23j . Both experimental 
interventions and biological evolution work in the bare 
parameter space, so this parameterization is fundamen- 
tal to the system, not an artifact of the modeling process. 
(2) Sloppiness depends not just upon the model, but also 
on the data it is fit to; exhaustive experiments designed 
to decouple the system and separately measure each pa- 
rameter will naturally not yield sloppy parameter sensi- 
tivities. (3) In biological systems fit to time-series data, 
Brown and Sethna [20 note that sloppiness may arise 
due to under-determined systems, proximity to bifurca- 
tions, and separation of time or concentration scales, but 
they doubt that these can explain all the sloppiness found 
in their model. Our analysis includes complete data on 
all species, and hence is overdetermined. Small eigenval- 
ues near bifurcations are associated with dynamic vari- 
ables, and not the system parameters we investigate. To 
study the effect of time and concentration scales we cal- 

2 

culated H x for a version of the Brown et al. model in 
which all concentrations and rate constants were scaled to 
one [47) . The resulting model remains sloppy, with eigen- 
values roughly uniformly spanning five decades (Support- 
ing Text S4). (4) Motivated by simple example systems, 
we have argued elsewhere that sloppiness emerges from 
a redundancy between the effects of different parameter 
combinations [33] . We are presently investigating decom- 
positions of parameter space into sloppy subsystems |46j 
and the use of physically or biologically motivated nonlin- 
ear coordinate changes to remove sloppiness or motivate 
simpler models. These potential methods for model re- 
finement, however, demand a complete and sophisticated 
understanding of the system that is unavailable for many 
biological systems of current interest. 

Parameter estimation has been a serious obstacle in 
systems biology modeling. With tens of unknown param- 
eters, a typical modeling effort might draw some values 



from the literature (possibly from in vitro measurements 
or different cell lines) [33j |38], set classes of constants to 
the same value (e.g., phosphorylation rates) [3"T1 13"2"l I3T] . 
and adjust key parameters to qualitatively best fit the 
existing data [3J[37J[3D]. In retrospect, these approaches 
may be successful because the models are sloppy — they 
can be tuned to reality by adjusting one key parame- 
ter per stiff direction, independently of how reliably the 
other parameters are estimated. 

Computational modeling is a potentially invaluable 
tool for extrapolating from current experiments and dis- 
tinguishing between models. But we cannot trust the 
predictions of these models without testing how much 
they depend on uncertainties in these estimated param- 
eters. Conversely, if we insist upon a careful uncertainty 
analysis, it would seem unnecessary to insist upon tight 
prior estimates of the parameters, since they do not sig- 
nificantly enhance model predictivity. Because the be- 
havior of a sloppy model is dominated by a few stiff di- 
rections that nonetheless involve almost all the param- 
eters, direct parameter measurements constrain predic- 
tions much less efficiently than comparably difficult ex- 
periments probing collective system behavior. 

Our suggestion of making predictions from models with 
very poorly known parameters may appear dangerous. A 
model with tens or hundreds of unmeasured parameters 
might seem completely untrustworthy; we certainly be- 
lieve that any prediction derived solely from a best-fit 
set of parameters is of little value. Uncertainty bounds 
derived from rigorous sensitivity analysis, however, dis- 
tinguish those predictions that can be trusted from those 
that cannot. Of course, successful fits and predictions 
may arise from models that are incorrect in significant 
ways; for example, one model pathway with adjusted pa- 
rameters may account for two parallel pathways in the 
real system. A model that is wrong in some details 
may nevertheless be useful in guiding and interpreting 
experiments. For computational modeling to be useful 
in incompletely understood systems, we must focus not 
on building the final, perfect, model with all parameters 
precisely determined, but on building incomplete, ten- 
tative and falsifiable models in the most expressive and 
predictive fashion feasible. 

Given that direct parameters measurements do not effi- 
ciently constrain model behavior, how do we suggest that 
experimentalists decide what experiment to do next? If 
the goal is to test the assumptions underlying a model, 
one should look for predictions with tight uncertainty es- 
timates that can be readily tested experimentally. If the 
goal is to reduce uncertainty in crucial model predictions, 
one must invoke the statistical methods of optimal exper- 
imental design, which we have studied elsewhere [27j and 
which may be conveniently implemented in modeling en- 
vironments that incorporate sensitivity analysis (such as 
SloppyCell gl]). 

In our approach, the model and its parameters can- 
not be treated in isolation from the data that informed 
model development and parameter fitting. This compli- 
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cates the task of exchanging knowledge in the modeling 
community. To support our approach, standards such 
as SBML [13] that facilitate automated model exchange 
will need to be extended to facilitate automated data ex- 
change. 

Every one of the 17 systems biology models we stud- 
ied exhibits a sloppy spectrum of parameter sensitivity 
eigenvalues; they all span many decades roughly evenly 
and tend not be aligned with single parameters. This 
striking and apparently universal feature has important 
consequences for the modeling process. It suggests that 
modelers would be wise to try collective parameter fits 
and to focus not on the quality of their parameter values 
but on the quality of their predictions. 



III. METHODS 



parameter sets consistent with the stated parameter un- 
certainties. (For each parameter 9i, the logarithm of its 
value is drawn from a normal distribution with mean 
log 9i and standard deviation cri og $ i specified by desired 
£.) For each parameter set, the Erk time course was cal- 
culated, and at each timepoint the shaded regions in the 
figure contain the central 95% of the time courses. 



D. Software 

All computations were performed in the open-source 
modeling environment Sloppy Cell, version 0.81 |44j . 
SBML files and SloppyCell scripts to reproduce all pre- 
sented calculations are in Dataset SI. 



A. Hessian Computations 

H* 2 can be calculated as 

H x 2 = _J_ V _J_ dys^O* ,t) dy s , c {0\t) 
j,k NcNs Z^ Tc<7 2 J q dlog6 . dlog9k ^ 

(3) 

Second derivative terms [d 2 y SyC (8* 7 t)/d\og 0i dlog 0j) 
might be expected, but they vanish because we evalu- 
ate H x ~ at 9*. Equation [3j is convenient because the 
first derivatives dy SiC (9* ,t) /d\og9k can be calculated by 
integrating sensitivity equations. This avoids the use 
of finite-difference derivatives, which are troublesome in 
sloppy systems. 

The projections Pi of the ellipsoids shown in Fig- 
ur e [2|V onto b are parameter axis i are proportional to 

,/( inv H* 2 ). ^ The intersections Ii with axis i are pro- 
portional to y 1 / Hf i , with the same proportionality con- 
stant. 

B. Parameter Uncertainties 

2 

To rescale H x so that it corresponds to fitting Nd 
data points, each with uncertainty a fraction / of the 
species' maximal value, we multiply H x by N d /f 2 . In 
the quadratic approximation, the one-standard-deviation 
uncertainty in the logarithm of parameter 9i after such 
a collective fit is given by of og0 . = (f 2 /N d ) (invijx 2 )^. 

The relative size of the 95% confidence interval is then 
E 4 = exp (4er loges ) - 1. 

C. Prediction Uncertainties 

The red and blue prediction uncertainties shown in 
Figure |4j3 were calculated by randomly generating 1000 



IV. SUPPORTING INFORMATION 

Text SI: Stiffest Eigenvectors 

Text S2: Effect of Other Poorly Determined Parame- 
ters 

Text S3: Fragility of Other Predictions 
Text S4: Rescaled Model of Brown et al. 
Text S5: Eigenvalue Analysis of Brodersen et al. Bind- 
ing Studies 

Dataset SI: SBML Files, SloppyCell Scripts, and x 2 - 
Hessians 



A. Accession Numbers 



Models discussed that 


are 


in the BioMod- 


els 


database !;42j are: 


(a) 


BIOMD0000000005, 


(c) 


BIOMD0000000003, 


(d) 


BIOMD0000000035, 


(e) 


BIOMD0000000002, 


(f) 


BIOMD0000000010, 


(h) 


BIOMD0000000021, 


(i) 


BIOMD0000000033, 


(k) 


BIOMD0000000022, 


(1) 


BIOMD0000000055, 


(n) 


BIOMD0000000015, 


(0) 


BIOMD0000000051, 



(p) BIOMD0000000056, (q) BIOMD0000000049. 
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